METHOD AND APPARATUS FOR DETECTING MOVING OBJECTS 
IN VIDEO CONFERENCING AND OTHER APPLICATIONS 



Field of the Invention 

5 The present invention relates generally to the field of image 

signal processing, and moire" particularly to techniques for 
detecting moving persons or' other objects of interest in a video 
signal or other type of image signal. 



10 Background of the Invention 

Detection and tracking of a person or other object of interest 
is an important aspect of video-camera-based systems such as video 
£3 conferencing systems, video surveillance and monitoring systems, 
^ and human-machine interfaces. For example, in a video conferencing 
K system, it is often desirable to frame the head and shoulders of a 
fii particular conference participant in the resultant output video 
j'j signal, while in a video surveillance system, it may be desirable 
= to frame the entire body of, e.g., a person entering or leaving a 
1° restricted area monitored by the system. 

?J) Conventional techniques for detecting persons in the above- 

= f noted applications include background subtraction, face detection 
q and skin tone detection. A significant problem with these and 
other conventional detection techniques is that many of them use 
general models or scene assumptions that make the techniques 
25 difficult to adapt for use with typical home or office scenes. For 
example, since moving persons in such home or office scenes often 
interact with objects such as chairs, sofas, and items on tables, 
a background subtraction technique would need to either keep track 
of a large number of foreground objects or update the background 
30 model frequently. In addition, face detection and skin tone 
detection techniques usually only handle only a limited number of 
head poses, e.g., faces close to the camera. Another problem with 
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these techniques is that the models used may be unable to 
accurately account for varying lighting conditions. Moreover, 
given the degree of clutter in typical home or office scenes, these 
and other conventional techniques that make use of edge detection 
5 are generally not suitable for real-time implementation. 

Other detection techniques make use of motion as a visual cue 
for scene analysis. Motion-based detection is particularly useful 
for cluttered scenes in which frequent movements of people and 
objects are common, such as the above-noted typical home or office 

10 environments. One known motion-based detection technique uses 
optical flow fields to group together as an object connected pixels 

f _ undergoing similar motion. Image differencing techniques utilize 

%2 the differences between consecutive frames to locate moving objects 

" in a scene. 

335 Such techniques are generally well suited for use in 

\i applications in which the number of moving objects in the scene is 
13 small and the interactions between them are limited. Although 
image differencing techniques can in some cases be adaptive to 
M dynamic environments, such techniques nonetheless generally do a 

ru 

3D poor job of extracting all relevant feature pixels. Another 
*3 significant problem with image differencing and other conventional 
" 3 motion-based detection techniques is that it can be hard to 
separate multiple moving objects occupying the same image region. 
An example of a conventional motion -based detection technique 
25 - is described in European Patent Application No. 635983 A2, entitled 
"Method and Means for Detecting People in Image Sequences . " This 
technique starts with analyzing a difference image, obtained by 
subtracting a current frame from a previous frame, and tries to 
locate the head positions of people in the image. However, this 
30 technique relies on the computation of continuous curve and 



2 




700806 

curvature extrema, and is therefore sensitive to noise and is 
computationally expensive. 

As is apparent from the above, a need exists for improved 
techniques for detecting persons in image processing systems such 
5 as video conferencing systems, video surveillance and monitoring 
systems, and human-machine interfaces. 

Summary of the Invention 

The invention provides methods and apparatus for detection of 
10 persons or other objects of interest in a video signal or other 

type of image signal. 
f ^ In accordance with an illustrative embodiment of the 

*,3 invention, a processing system generates, e.g., a thresholded 
~Z difference image by processing an image signal received from a 
§3 camera. The difference image is then segmented into regions 
i_q bounded by vertical lines passing through the image, and silhouette 
12 candidates are identified in one or more of the regions. Tensor 
^j, voting is used to determine saliency values and corresponding 

tangents for each of the silhouette candidates, and the resulting 

ru 

20 values and tangents are used to detect the object of interest. If 

'3 the object of interest is a moving person, a neck position of the 
moving person may be detected by analyzing a sum of x- components of 
tangents along a corresponding silhouette. The detected neck 
position may then be utilized to determine a head position and a 

25 head size for the moving person. The present invention thus 
provides a real-time processing system that is particularly well 
suited for detecting moving persons or other objects of interest in 
a typical home or office scene, or other cluttered environment. 

The detection techniques of the present invention provide a 

30 number of significant advantages over conventional techniques. For 
example, detection techniques in accordance with the invention are 
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generally more robust to noise and less computationally expensive 
than conventional techniques. In addition, the techniques of the 
invention can detect moving persons and other objects of interest 
in real time, and can handle the presence of multiple persons, and 
5 varying body sizes and poses. The number of moving objects, 
together with positions and silhouettes of the objects, can be 
computed automatically without any initialization. Moreover, the 
invention significantly improves the efficiency and accuracy of the 
detection of particular features, such as the neck and head of a 
10 moving person. The techniques of the invention can be used in a 
wide variety of image processing applications, including video- 
camera-based systems such as video conferencing systems, video 

Q 

i_3 surveillance and monitoring systems, and human -machine interfaces. 
= J These and other features and advantages of the present invention 
will become more apparent from the accompanying drawings and the 

!^ following detailed description. 

vi 

Q 

[. Brief Description of the Drawings 

H FIG. 1 is a block diagram of an image processing system in 

fil 

20 which the present invention may be implemented. 

\3 FIG. 2 is a flow diagram showing the operation of an exemplary 

* J moving object detection technique in accordance with an 
illustrative embodiment of the invention. 

FIGS. 3A and 3B show an example of two consecutive frames of 
25 a typical cluttered office scene that is used to illustrate the 
detection techniques of the present invention. 

FIG. 4 shows a thresholded difference image generated from the 
frames of FIGS. 3A and 3B. 

FIG. 5 illustrates the segmentation of the FIG. 4 difference 
30 image into regions. 
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FIGS. 6 and 7 show silhouette candidates and corresponding 
saliency values, respectively, as computed for the segmented 
difference image of FIG. 5. 

FIG. 8 shows silhouettes extracted from the silhouette 
5 candidates of FIG. 6. 

FIG . 9 illustrates a head and shoulder silhouette analysis 
model that may be applied to the extracted silhouettes of FIG. 8. 

FIG. 10 shows detected neck and head positions for the FIG. 4 
difference image. 



Detailed Description of the Invention 

f _ FIG. 1 shows an image processing system 10 in which moving 

%2 object detection techniques in accordance with the invention may be 
" implemented. The system 10 includes a processor 12, a memory 14, 
£5 an input/output (I/O) device 15 and a controller 16, all connected 



C3 includes a camera 18 that is coupled to the controller 16 as shown. 
l A The camera 18 may be, e.g., a pan-tilt-zoom (PTZ) camera, a zoom 
f* camera or any other suitable image capture device. The term 
= % "camera" as used herein is therefore intended to include any type 
'%3 of image capture device. 

The system 10 may be adapted for use in any of a number of 

different image processing applications, including, e.g., video 

conferencing, video surveillance, human-machine interfaces, etc. 
25 More generally, the system 10 can be used in any application which 

can benefit from the improved detection capabilities provided by 

the present invention. 

Elements or groups of elements of the system 10 may represent 

corresponding elements of an otherwise conventional desktop or 
30 portable computer, as well as portions or combinations of these and 

other processing devices. Moreover, in other embodiments of the 
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to communicate over a system bus 17 



The system 10 further 
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invention, some or all of the functions of the processor 12, 
controller 16 or other elements of the system 10 may be combined 
into a single device. For example, one or more of the elements of 
system 10 may be implemented as an application specific integrated 
5 circuit (ASIC) or circuit card to be incorporated into a computer, 
television, set-top box or other processing device. 

The term "processor" as used herein is intended to include a 
microprocessor, central processing unit, microcontroller or any 
other data processing element that may be utilized in a given data 

10 processing device. In addition, it should be noted that the memory 
14 may represent an electronic memory, an optical or magnetic disk- 
based memory, a tape-based memory, as well as combinations or 

i3 portions of these and other types of storage devices . 

" The present invention in an illustrative embodiment provides 

11> a motion -based detection technique that makes use of temporal 

12 differencing and perceptual organization principles to detect 
q moving persons or other objects of interest in cluttered indoor 
f scenes such as home and office environments. 

U In general, the illustrative embodiment of the invention 

extracts motion silhouettes from a temporal difference image that 

13 has been binarized by thresholding. It first analyzes the 
%=i distribution of non-zero pixels to segment the scene into regions. 

For each region , the non-zero _p_ixels_c losest to a region bounding 
box form the first approximation of the motion silhouette. To deal 

25 with noise and outliers, the saliency of the silhouette points are 
then evaluated by examining the configuration of the silhouette 
using tensor voting. Once the salient points on the silhouettes 
are identified, missing data, together with the complete 
silhouette, can be inferred by applying tensor voting again. 

30 Alternatively, the inferred geometric properties of the salient 



6 




700806 

silhouette points can be used to locate the head position, e.g., so 
as to verify that the moving object is human. 

FIG. 2 is a flow diagram showing the motion-based detection 
technique of the illustrative embodiment. For this embodiment, it 
5 is assumed that the objects of interest include one or more persons 
in a scene corresponding to a cluttered environment such as an 
indoor home or office scene. FIGS. 3A and 3B show consecutive 
frames t-1 and t, respectively, for an example of a cluttered 
office scene of this type, with two moving persons to be detected. 

10 Relevant information to be detected in this example includes the 
head position and head size of each of the persons. In the 

^ following description, an overview of the detection technique will 

11 be provided with reference to FIG. 2, and then each of the steps of 
= z the technique will be described in greater detail. 

15 In step 20 of FIG. 2, a thresholded difference image is 

| = generated for the scene to be analyzed. As previously noted, such 
£3 a difference image can be generated in a conventional manner by 
taking the difference between two frames or other images of the 
U scene of interest, and then applying an appropriate thresholding 
2f) technique. FIG. 4 shows an example of a thresholded difference 
□ image generated for the cluttered office scene using the two 
"~ consecutive frames of FIGS. 3A and 3B. 

In step 22, the scene of interest is segmented into regions by 
analyzing a horizontal projection profile ffP(x) to be described in 
25 greater detail below. FIG. 5 illu strates the segmentation process 
as applied to the difference image of FIG. 4. It can be seen that 
the segmental: loir^prolTes's Tn **th"Ts" "ex^pl^~divTde's~"th'e~dlf f erence 
image into a total of five regions separated by four— ver-tical 
linesT' These regions include regions 50 and 52", each" of which 
3 0 corresponds to one of the persons to be detected". ~~- 
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In step 24, silhouette candidates in the segmented image are 
identified. Saliency values for the silhouette candidates are then 
determined in step 26 using a tensor voting technique to be 
described in greater detail below. In step 28, the neck of a 
person corresponding to a given silhouette is detected by analyzing 
the sum of the x-component of tangents along the silhouette. The 
detected neck position is then used in step 30 to determine the 
head size and head position of the moving person. 

The motivations behind the detection technique of FIG. 2 will 
now be described in greater detail, followed by specific 
description of particular steps in the process. 

In accordance with the well-known constant brightness 
constraint, when movements in a given scene are small, frame 
differencing produces effects similar to a typical response to an 
edge detector. This follows from the constant brightness constraint 
equation: 



ru 

2=0 where, given an image sequence I(x,y,t) , I r = , x f = — 
.= dt dt 



dy dl dl 

y t = — , I = — , and J„ = — . In words, no or low changes in 
dt dx y dy 

intensity value can be caused by no motion or low contrast in the 

local neighborhood, or both. However, for segmentation purposes, 

it is desirable to determine a true change map C(x,y): 



[| [xt yt] | > 0 1 
C(x, y) = < 

[otherwise 0 
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from which moving objects can easily be extracted when the camera 
is stationary . 

Notice that despite the limited amount of information in the 
5 difference image of FIG. 4, humans can still segment the underlying 
objects and "fill in" the interior of the objects where no 
information is given, i.e., the human vision system can essentially 
compute the change map C(x,y) from a difference image I t (x,y) . As 
will be described below, the present invention exploits perceptual 
10 organization principles that may have been used in the derivation 
of C(x,y) in human vision system, to obtain an abstraction of 
?1 C(x,y) from the difference image I t (x,y) . 

%3 The first grouping principle exploited by the present 

°Z invention is the attentive nature of the perception of human shape . 
£5 An attentive process is one that needs to use previously acquired 
|5 knowledge in order to perceive an object in an image, which usually 
13 takes longer than a pre-attentive process. For example, when a 
[i difference image such as that of FIG. 4 is turned upside down, the 
H perception of human bodies in the scene is weaker, especially for 
ib the person in the back. We usually interact with people in an 
w upright position, and therefore are less familiar with the shape of 
body parts of a person in an upside-down position. 

In order to use this prior knowledge, the detection technique 
of the illustrative embodiment assumes people appear in an upright 
25 position in the image sequence. Consequently, just head and 
shoulder silhouettes capture enough information to describe people 
in the scene. It is this abstraction of C(x,y) that the detection 
technique in the illustrative embodiment seeks to obtain in order 
to describe moving persons in a cluttered indoor scene. The 
3 0 description should generally include the number of people detected, 
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the image coordinates of each person, and the motion silhouette of 
each person . 

If C(x,y) were given, the extraction of motion silhouettes 
would be straightforward. For each object, the silhouette is 
defined by points closest to the bounding box of the object. 
However, we do not have access to C(x,y). Instead, we observe that 
the silhouettes are reasonably preserved in the difference image 
lt(x,y), which arguably is a degraded version of C(x,y) . We can 
therefore extract the motion silhouettes from I t (x,y) directly. As 
the intensity values of l t (x,y) depend mostly on local contrast, 
they are not relevant to motion detection. We can therefore apply 
a thresholding technique which takes into account sensor noise to 
produce a degraded version of C(x,y) . An example of a thresholding 
technique suitable for use with the present invention can be found 
in P.L. Rosin, "Thresholding for Change Detection," Proc . ICCV, Jan 
1998, pp. 274-279. 

The segmentation approach used in the illustrative embodiment 
of the present invention makes use of a grouping principle for pre- 
attentive perception. This principle is similar to the "matter is 
cohesive" principle, described in D. Marr, "Vision: A Computational 
Investigation into the Human Representation and Processing of 
Visual Information," W.H. Freeman and Co., San Francisco, 1982, 
which is commonly interpreted as a smoothness or continuity con- 
straint . 

This constraint has been used by various techniques that 
produce a non -parametric description for the segmentation and 
grouping of curves and/or regions from noisy data set. One such 
technique, the above-noted tensor voting technique, uses second- 
order symmetric tensors for data representation and non-linear 
voting for data communication, and handles missing data, outliers, 
and discontinuities in an efficient and effective way. It requires 

10 
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no initialization, and the only two parameters are scale and 
subject threshold. As previously noted, step 26 of the FIG. 2 
process utilizes tensor voting to determine saliency values for 
silhouette candidates. 

The region segmentation step 22 of the FIG. 2 process will now 
be described in greater detail. In order to implement such a 
process effectively in real time, the invention exploits the above- 
noted observation that humans usually appear in upright positions 
in an image. As a result, segmenting the scene into regions using 
vertical lines as" shown in FIG. 5 is sufficient for most 



applications. It may be necessary~to apply"" the" segmentation step 



more" - than once^ e.g.., if a scene -is -under- segmented such that a 
given regig ji_conJ:ains_ more-,than .one -silhouette I 

As previously noted, the region segmentation step 22 segments 
the scene into regions by analyzing the horizontal projection 
profile HP(x) of the m x n binarized difference image J t (x, y) , 
where : 



HP{x) = £" =i It{x,j) 



The task here essentially is to locate where discontinuities occur. 
Based on an approach described in M-S. Lee and G. Medioni, 
"Grouping .,-,->, O-, into Regions, Curves, and Junctions," Computer 
Vision and Image Understanding, vol. 76, No. 1, Oct. 1999, pp. 54- 
69, the region boundary saliency RS(x) is defined as: 

RS(x) = £ w{x - i)ffP(i) - X w(i - x)HP(i) 
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where w(x) = e a * is the Gaussian decay function. A value of o = 10 
is used in the examples shown herein, but other values could also 
be used. RS(x) measures the horizontal difference of data density 
in the local neighborhood. The maxima in \RS(x)\, which correspond 
to locations where maximal change in data density occur, are 
candidates for region boundaries. The labels of the identified 
region boundaries are given by the sign of the corresponding RS 
values, where negative RS indicates a left boundary and positive RS 
indicates a right boundary. The ordering of the boundary labels, 
and the actual differences in HP, are used to eliminate false 
region boundaries in the candidate set. 

Once the scene is segmented into regions, points closest to 
the region bounding boxes are identified as candidates on the 
silhouettes, as noted in step 24 of FIG. 2. FIG. 6 shows the 
silhouette candidates. Note that the bottom part of a given 
silhouette is not useful and therefore is not extracted. 

In order to eliminate outliers and to deal with noise, the 
saliency of each candidate point is assessed by analyzing the 
silhouette configuration in the local neighborhood using a tensor 
voting technique, as indicated in step 26 of FIG. 2. A tensor 
voting technique suitable for use with the present invention is 
described in M-S. Lee and G. Medioni, "Grouping .,-,->, O-, into 
Regions, Curves, and Junctions," Computer Vision and Image Under- 
standing, Vol. 76, No. 1, Oct. 1999, pp. 54-69. For the i th 
candidate point (p if g±) , the saliency tensor S(Pi,qi) is computed 
as : 
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5 d(i,j) is the distance between the i th candidate and the j th 
candidate, and w(x) is the Gaussian decay function. Since w(x) has 
a limited extend, with o=6 in the current system, the computation 
of S can be efficiently implemented by convolution with tensors. 
f]j The saliency tensor S(pi_,qi) is then decomposed into its eigenvalues 
ri) -2 mai , A min , and eigenvectors e max , e mln . The saliency of the i th 
" candidate as a silhouette point is defined to be A max — A mln , with 
i"U tangent estimated to be e max . FIG . 7 depicts the saliency values so 

In 

E3 computed for the silhouette candidate points of FIG. 6. 
j\ In applications in which complete silhouettes are desirable, 

B#5 one can infer missing data and group points on the same silhouette 

"Z together. This can be achieved by applying tensor voting to every 

ii point in the image to assess its saliency as a silhouette point. 

*~ For a point (Xi, yij, the saliency tensor S(xi,yi) is defined to be: 



S(x if yj = 



X^ 



where [ u! l} v' lj ] is a vector vote generated by the j' th candidate point 
for the point (Xi,y L ). One can use a vote generation function that 
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is independent of the location and the orientation of the voting 
candidate, such that a voting kernel, with o = 10 , as described in 
G. Guy and G. Medioni, " Inferring Global Perceptual Contours from 
Local Features", IJCV, Vol. 20, No. 1/2, 1996, pp. 113-133. To 
5 generate votes, the voting kernel is aligned with the tangent in- 
ferred for the voting candidate, and centered at the corresponding 
location. Each vote contribution is further weighted by the 
saliency of the voting candidate. Additional details regarding the 
tensor voting process can be found in the above-cited M-S. Lee and 
10 G. Medioni reference. 

After all the candidates have cast their votes, the saliency 
^ tensor so obtained is decomposed into its eigensystem and the 
W saliency of the point is determined to be A max - A nln . FIG. 8 shows 
_= a dense saliency map computed for the saliency values of FIG. 7. 
fif> Salient silhouettes can be extracted from the saliency map of FIG. 
m 8 by locating maximal curves using a marching procedure such as 
u that described in C-K Tang and G. Medioni, "Inference of Integrated 
U Surface, Curve, and Junction Descriptions From Sparse 3D Data", 
Jf; IEEE Trans. PAMI, vol. 20, no. 11, 1998, pp. 1206-1223. 
233 In the illustrative embodiment, the saliency and orientation 

'= information computed for the silhouette candidates is used to 
determine the head position and head size of the corresponding 
moving person . The change in orientation along a head and shoulder 
silhouette is most drastic near where the neck is located. This 
25 can be illustrated by the simplified head and shoulder model shown 
in FIG. 9. 

The x- components of the tangents along a given silhouette form 
two discontinuous clusters, one at the top of the head, and one at 
the neck position. This is similar to the above-described maximal 
30 detection for region segmentation, and can be solved using similar 
techniques. The neck position is detected first, as indicated in 
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step 28 of FIG. 2. Once the neck position is detected, candidate 
points above the neck are used to determine the head position and 
size, as indicated in step 30 of FIG. 2. 

FIG. 10 shows the detected neck positions, head positions and 
5 head sizes for the moving persons in the original example scene of 
FIGS. 3A and 3B. Notice that by analyzing the shape of the 
silhouette, one can eliminate non-human shape silhouettes, such as 
those of a chair, desk, computer, etc. Once a moving person or 
other object is detected, conventional tracking techniques can then 
10 be used to track further movement of the object. Such techniques 
are well known in the art and therefore are not described in detail 
herein . 

C3 

,3 The above-described detection techniques of the present 

== invention provide a number of significant advantages over 
jij5 conventional techniques. For example, the techniques of the 
j= invention can detect moving persons and other objects of interest 
P in real time, and can handle the presence of multiple persons, and 
: varying body sizes and poses. The number of moving objects, 
' T ± together with positions and silhouettes of the objects, can be 
K) computed automatically without any initialization. The only 
% 2 parameters used in the illustrative embodiment are scale, which 
w determines the size of a neighborhood being used to establish sa- 
liency of data, and subjective threshold, which defines the minimum 
saliency of "significant" motion. 
25 The present invention thus provides a real-time image 

processing system that can detect moving persons or other objects 
of interest in cluttered indoor environments. Moreover, the 
extracted motion silhouettes provide useful shape information that 
when combined with other image cues, such as color, edges, and 
3 0 intensity, can be used to robustly track objects in a typical home 
or office scene, or other cluttered environment. 
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The above-described embodiments of the invention are intended 
to be illustrative only. For example, the invention can be used to 
implement real-time detection and tracking of any desired object of 
interest, and in a wide variety of applications, including video 
conferencing systems, video surveillance systems, and other camera - 
based systems . The invention can also be implemented at least in 
part in the form of one or more software programs which are stored 
on an electronic, magnetic or optical storage medium and executed 
by a processing device, e.g., by the processor 12 of system 10. 
These and numerous other embodiments within the scope of the 
following claims will be apparent to those skilled in the art. 
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